Most video coding standards including MPEG and H.26x families [1-3] employ quantization of transform blocks followed by entropy encoding of the quantization levels to reduce the number of bits to represent them. Those standards define only inverse quantization (Q−1), i.e., a way of mapping the quantization levels back to transform coefficients and leave the forward quantization (Q) to an encoder's choice. The most common schemes for quantizing the transform coefficient, TC, with a given quantization step size, stepSize, can be represented as in Eq. (1) and Eq. (2), respectively for the case of without and with dead zone around zero:level=Q(|TC|)=(|TC|+α*stepSize)/stepSize,  Eq. (1)level=Q(|TC|)=0, if |TC|<β*stepSize, otherwise (|TC|−β*stepSize+α*stepSize)/stepSize  Eq. (2)where “/” represents the integer division operation, α is to exploit the characteristic of the probability distribution of the transform coefficient, and β is introduced to make a dead zone around zero. In case of the uniform distribution of probability, all the levels have the same probability and so are encoded with the same number of bits. In this case, the best performance is shown at α=0.5. But the distribution of the transform coefficient tends to show Laplacian or Exponential distribution that is far different from the uniform one. To reflect this distribution, lower levels are usually represented in the less number of bits and therefore α is set to a number smaller than 0.5. β defines the size of dead zone and is given by the standards. The β is set to 0 in H.264 quantization since the standard doesn't employ a dead zone around zero [3, 4]. The two quantization schemes are also illustrated in FIG. 1A and FIG. 1B.
What is needed to achieve better coding performance is joint optimization of quantization and entropy coding. However, in many video coding systems including MPEG and H.26x families the entropy code tables and the inverse quantization process are already fixed, and the encoder is not allowed to change them. In this case, what the encoder needs to do is to optimize the quantization region for each level only. In such video coding systems, the quantization scheme is designed by using either one of the two functions in Eqs. (1) and (2) and selecting the value for α experimentally. This simple approach does not fully exploit the characteristic of the entropy coding system. What is needed is a quantization scheme that better exploits the entropy encoding scheme employed. What is also needed is a scheme that quantizes to zero all transform coefficients in blocks with small energy in the sense of rate-distortion optimization.